Discussing with your casemix specification with natural language: how to make Large Language Models (LLM) understand NordDRG logic?


Tapio Pitkäranta a

Introduction
Recent advancements in large language models (LLMs) have sparked significant interest across a broad spectrum of use cases and business domains [1,2,3]. This paper delves into the application of LLMs in interpreting the complex logic of the NordDRG system [4]. Our findings suggest that the current general LLMs struggle to effectively handle the NordDRG logic. However, when LLMs are enhanced with Retrieval Augmented Generation (RAG) [5], the enhanced NordDRG AI agent is capable of engaging in natural language conversations with various stakeholders. This paper provides a comparative analysis between the general LLM and the RAG-enhanced model, highlighting the latter's enhanced ability to address specific queries related to DRG logic. Such progress marks a significant stride in healthcare information systems, presenting a user-friendly interface for intricate medical coding systems.

The Nordic Diagnosis Related Groups (NordDRG) system plays a critical role in healthcare resource management. However, its intricate logic poses a challenge in understanding and navigation. Addressing this, our study explores the potential of AI, particularly LLMs, to simplify DRG logic interpretation through natural language processing.

Methods
We implemented a RAG AI model, where the NordDRG specification is indexed using a retrieval model. The LLM component serves two functions: it formalizes user queries into a structured format and synthesizes the retrieved information into coherent responses. This dual approach enables the RAG model to provide contextually relevant information in a conversational manner.

Results
The paper details example dialogues between various stakeholders LLM based AI Agents. These interactions showcase the AI's ability to understand complex queries and provide accurate, context-aware responses. The RAG model's performance was markedly better than the general LLM, especially in interpreting detailed DRG logic.

Discussion
Our findings indicate that while general LLMs struggle with the nuances of specialized medical coding system such as NordDRG, the RAG model's dynamic retrieval capability significantly enhances its performance. This improvement underscores the model's potential as a valuable tool for healthcare professionals, aiding in decision-making and administrative tasks.

Conclusions
Integrating RAG AI into the NordDRG system marks a significant advancement in making medical coding systems more accessible. This technology promises to streamline administrative processes and support healthcare professionals in managing complex coding tasks. Future research should focus on refining the AI model and exploring its application in other specialized medical systems.


References
  1. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Kaiser, and I. Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
  2. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
  3. T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020
  4. NordDRG casemix system: https://nordcase.org/
  5. arXiv:1810.04805, 2018. 7. K. Guu, K. Lee, Z. Tung, P. Pasupat, and M. Chang. Retrieval augmented language model pre-training. In International conference on machine learning, pages 3929 - 3938. PMLR, 2020

a PhD / Aalto University, Finland

Original Version in PDF